Towards a standard evaluation method for grammatical error detection and correction
نویسندگان
چکیده
We present a novel evaluation method for grammatical error correction that addresses problems with previous approaches and scores systems in terms of improvement on the original text. Our method evaluates corrections at the token level using a globally optimal alignment between the source, a system hypothesis, and a reference. Unlike the M Scorer, our method provides scores for both detection and correction and is sensitive to different types of edit operations.
منابع مشابه
Human Evaluation of Grammatical Error Correction Systems
The paper presents the results of the first large-scale human evaluation of automatic grammatical error correction (GEC) systems. Twelve participating systems and the unchanged input of the CoNLL-2014 shared task have been reassessed in a WMT-inspired human evaluation procedure. Methods introduced for the Workshop of Machine Translation evaluation campaigns have been adapted to GEC and extended...
متن کاملBetter Evaluation for Grammatical Error Correction
We present a novel method for evaluating grammatical error correction. The core of our method, which we call MaxMatch (M), is an algorithm for efficiently computing the sequence of phrase-level edits between a source sentence and a system hypothesis that achieves the highest overlap with the goldstandard annotation. This optimal edit sequence is subsequently scored using F1 measure. We test our...
متن کاملGrammatical Error Correction with Neural Reinforcement Learning
We propose a neural encoder-decoder model with reinforcement learning (NRL) for grammatical error correction (GEC). Unlike conventional maximum likelihood estimation (MLE), the model directly optimizes towards an objective that considers a sentence-level, task-specific evaluation metric, avoiding the exposure bias issue in MLE. We demonstrate that NRL outperforms MLE both in human and automated...
متن کاملThere's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction
Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing grammatical edits that are correct but not in the gold standard. We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metr...
متن کاملThe Impact of Immediate Grammatical Error Correction on Senior English Majors’ Accuracy at Hebron University
This study aimed at investigating the effects of grammatical error correction on EFL learners’ accuracy. Twenty-two male and female senior students were chosen randomly to respond to a questionnaire investigating their beliefs about immediate grammatical error correction. Actually, the study was conducted in order to answer this question: what is the effect of grammatical error feedback on stu...
متن کامل